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Abstract 

This  investigation  sought  to  determine  how  in-house  research 
projects  are  selected  at  the  U.S.  Air  Force  Wright  Aeronautical 
Laboratories  (AFWAL).  The  problem  was  explored  by  studying  ten  aspects 
of  the  research  project  selection  process. 

This  analysis  was  accomplished  by  using  a  combination  of  personal 
and  telephone  interviews.  Ten  individuals  from  each  of  the  four  AFWAL 
Laboratories  were  interrviewed.  The  results  illustrated  that  few  of 
the  respondents  used  a  formal  decision  method  model  when  selecting 
research  projects.  Most  of  the  in-house  projects  selected  at  AFWAL  are 
chosen  via  a  consensus  of  agreement  between  the  various  levels  of 
management  in  each  laboratory. 
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RESEARCH  AND  DEVELOPMENT  PROJECT  SELECTION  METHODS 
AT  THE  AIR  FORCE  WRIGHT  AERONAUTICAL  LABORATORIES 


I.  Introduction 


Background 

With  the  emphasis  the  Reagan  administration  has  placed  on  military 
spending,  funding  for  research  and  development  (R&D)  in  both  the  Air 
Force  and  the  private  sector  is  expected  to  increase.  R&D  funding  is 
already  a  significant  part  of  the  budget  for  both  the  Air  Force  and 
many  corporations  in  the  private  sector.  Therefore,  R&D  project 
management  efficiency  must  be  maximized  to  insure  that  these  funds 
are  managed  effectively. 

The  nature  of  R&D  in  private  industry  is  similar  in  many  ways  to 
R&D  conducted  in  a  military  environment.  A  study  of  the  problems  in 
both  arenas  could  yield  insight  into  ways  of  increasing  R&D  management 
efficiency  in  Air  Force  laboratories. 

A  crucial  and  difficult  decision  for  laboratory  managers  is  the 
proper  selection  of  research  projects  for  their  laboratory  project 
portfolios.  While  there  have  been  many  studies  conducted  examining 
this  process  in  corporate  laboratories,  few  studies  of  this  nature 


have  been  conducted  in  Air  Force  laboratories 


General  Issue 


How  can  the  selection  of  research  projects  for  Air  Force 
laboratory  project  portfolios  be  improved?  Research  project  portfolio 
selection  is  crucial  to  Air  Force  laboratory  managers  for  three 
reasons.  First,  their  decisions  will  influence  technology  development 
that  may  be  incorporated  into  new  weapons  systems.  Second,  these 
managers  seek  to  optimize  limited  resources  in  competition  with 
numerous  projects.  Third,  the  selection  of  research  projects  is  an 
important  function  of  laboratory  management.  To  accomplish  this  task 
a  manager  must  trade  off  and  prioritize  candidate  projects  so  they 
match  available  resources  and  organizational  objectives. 

Specific  Problem 

How  do  Air  Force  Laboratory  managers  presently  select  research 
projects  for  their  laboratory  project  portfolios?  Research  project 
proposals  come  to  laboratory  managers  from  a  variety  of  sources.  On 
one  end  of  the  spectrum  are  external  source  research  projects  that  come 
attached  with  an  appropriation  from  upper  levels  of  the  Air  Force  or 
DOD.  At  the  opposite  end  are  those  research  proposals  originated  by 
laboratory  researchers  that  require  laboratory  funds.  This  research 
effort  focuses  on  research  projects  that  laboratory  managers  have  the 
discretion  of  selecting  or  rejecting. 

Research  Objectives 

Ten  aspects  of  the  research  project  selection  process  at  the 
Air  Force  Wright  Aeronautical  Laboratories  (AFWAL)  were  explored. 
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This  section  delineates  those  research  objectives  and  describes  the 
rationale  for  exploring  them. 

1.  Research  Project  Selection  Factors.  Before  selecting  a 
project  a  manager  must  consider  the  attributes  of  a  project  and  compare 
them  to  research  goals.  These  attributes  will  be  the  inputs  into  any 
decision-making  process  the  manager  uses.  Therefore,  the  first 
research  objective  was  to  gather  data  concerning  project  selection 
factors  used  in  AFWAL. 

2.  Formal  Decision  Method  Use .  The  second  objective  was  to 
determine  the  extent  that  AFWAL  researchers  and  managers  use  formal 
decision-making  methods  in  selecting  research  projects.  Patterns  of 
the  use  of  such  methods  in  the  various  laboratory  organizations  were 
explored. 

3.  Formal  Decision-Making  Techniques.  The  third  research 
objective  focused  on  decision-making  techniques  now  being  used  at 
AFWAL.  Inputs  for  this  question  were  generated  from  information 
gathered  during  a  literature  review  of  this  subject. 

The  effects  of  a  management  science/operations  research  background 
were  also  evaluated  since  operations  research  plays  a  prominent  role  in 
decision-making  theory  and  practice.  A  manager  who  is  familiar  with 
this  discipline  may  be  inclined  to  apply  some  of  it  to  the  research 
project  selection  process. 

4.  Decision-Making  Technique  Awareness .  The  fourth  objective  was 
to  determine  what,  if  any,  decision-making  methods  the  participants  of 
this  study  were  cognizant  of  but  did  not  use.  The  objective  was  to 


gather  insights  into  the  viability  or  impracticality  of  the  various 
decision  techniques. 

5.  Respondent's  Desires  to  Change  Methods.  The  fifth  objective 
sought  to  determine  the  degree  to  which  the  respondents  were  satisfied 
or  dissatisfied  with  their  present  way  of  selecting  research  projects. 
If  the  respondents  were  dissatisfied,  their  recommended  solutions  for 
improving  the  process  were  sought. 

6.  Formal  Laboratory  Management  Training.  The  sixth  objective 
was  to  identify  any  relationship  that  may  exist  between  the  knowledge 
gained  from  specific  laboratory  management  training  courses  and  their 
applications  to  research  project  selection. 

7.  Manager 's  Budget  vs  Formal  Decision  Methods.  The  seventh 
research  objective  focused  on  how  the  size  of  a  manager's  budget  would 
impact  his  inclination  to  use  a  formal  decision  method.  The  assumption 
entering  this  research  was  that  the  larger  the  budget  a  manager 
controls,  the  higher  the  probability  that  manager  will  use  some  formal 
decision  method. 

8.  Project  Select ion/ Assignment  vs  Formal  Decision  Method  Use. 

The  eighth  objective  was  to  see  if  the  selection  and/or  assignment 

volume  of  an  individual  influences  that  person's  propensity  to  use 

a  formal  decision  method.  The  underlying  hypothesis  here  was  that 

* 

a  heavier  volume  of  project  selections  and/or  assignments  would  by 
necessity  make  using  such  a  method  more  attractive. 

9.  Selection  Method  Propensity  vs  Project  Value.  The  ninth 
research  objective  was  to  determine  if  a  project's  dollar  value 
directly  influences  the  formal  decision  method  used.  The  initial 


hypothesis  was  that  the  higher  the  dollar  value  of  a  project,  the  more 
likely  that  project  should  be  selected  using  a  formal  decision-making 
method. 

10.  Project  Portfolio  Selection.  The  final  research  objective 
focused  on  whether  different  techniques  were  used  for  single  projects 
versus  portfolio  projects.  Many  commercial  laboratories  treat  the 
function  of  selecting  research  project  portfolios  (several  projects 
to  meet  a  single  research  objective)  differently  than  selecting  a 
single  research  project.  The  purpose  of  researching  this  area  was  to 
determine  if  AFWAL  also  conducted  this  selection  process  differently. 


II.  Literature  Review 


Introduction 

This  literature  review  covers  the  following  areas:  previous  survey 

studies  in  R&D  project  selection,  a  survey  of  some  specific  R&D  project 

selection  model  research,  and  a  review  of  some  common  selection  methods 

that  are  used  in  selecting  R&D  projects. 

I 

I 

Previous  Survey  Studies  in  R&D  Project  Selection 

Baker  and  Pound ,  1964.  This  survey  was  conducted  to  confirm 
two  observations  that  the  two  researchers  had  concerning  R&D  project 
selection  methods.  The  first  observation  was  that,  despite  the  fact 
that  many  researchers  had  published  papers  on  R&D  project  selection, 
few  had  published  more  than  one  or  two  papers.  The  second  observation 
was  that  very  few  of  the  decision  methods  proposed  in  these  papers 
were  used  (4:124). 

Baker  and  Pound  used  an  extensive  literature  review  and  a 
combination  of  interviews  and  written  survey  techniques  with  a  sampling 
of  laboratory  managers  from  a  variety  of  laboratories.  Many  of  the 
managers  were  present  at  a  particular  seminar.  The  models  examined 
during  the  course  of  the  literature  review  were  used  as  inputs  for 
the  surveys  and  interviews  (4:124). 

Based  on  the  responses  received  from  the  interviews  and  surveys, 
Baker  and  Pound  made  the  following  conclusions:  1)  there  has  been 


insufficient  testing  of  many  of  the  R&D  selection  models,  2)  objectives 
and  criteria  of  R&D  projects  are  insufficiently  clear  to  make  good  use 
of  many  of  the  models,  and  3)  the  models  did  not  adequately  deal  with 
the  variable  of  technical  uncertainty  (4:130-131). 

Cetron,  Martino  and  Roepcke,  1967.  The  researchers  performed  a 
literature  review,  which  presented  approximately  thirty  methods  that 
were  used  for  quantitative  evaluation  and  selection  of  R&D  projects. 
Each  method  was  "compared  and  contrasted  with  each  other  relative  to  a 
standard  set  of  features  which  they  may  possess,  to  a  standard  set  of 
characteristics  relating  to  ease  of  use,  and  to  scientific  or 
technological  areas  of  applicability"  (7:4). 

The  features  of  the  methods  that  were  analyzed  were: 

1.  Utility  Measure  -  utility  or  success  value  of  the 
R&D  project. 

2.  Probability  of  success 

3.  Orthogonality  of  Criteria  -  the  fact  that  certain 
criteria  may  be  mutually  exclusive. 

4.  Sensitivity 

5.  Rejected  Alternatives  Retention  -  retaining  a  project 
that  has  previously  been  rejected  due  to  a  funding 
limitation. 

6.  Classification  Structure  -  relationship  between  R&D 
project  and  hierarchy  of  organizational  goals. 

7.  Time 

8.  Strategies 

9.  System  Cross  Support 

10.  Technology  Cross  Support 

11.  Graphical  Display 


7 


12.  Flagging  -  pointing  up  problem  areas 

13.  Optimization  Criteria 

14.  Constraints 

13.  Degree  of  Computerization  Required 

The  scientific  and  technological  areas  of  applicability  that  were 
examined  were: 

1.  Research 

2.  Exploratory  Development 

3.  Advanced  Development 

4.  Engineering 

The  researchers  drew  the  following  conclusions: 

Each  method,  within  its  capabilities  and  limitations,  can 
provide  assistance  to  the  management  of  an  R&D  enterprise  in 
appraising  the  worth  of  its  R&D  effort.  In  particular,  the 
use  of  quantitative  methods  tends  to  eliminate  bias,  provide 
a  degree  of  consistency,  and  force  managers  to  render  their 
judgements  more  explicit  in  evaluating  R&D  programs.  [7:10] 


The  methods  were,  however,  limited  by  two  factors  —  the  validity 
of  the  information  inputs  by  laboratory  personnel  and  higher  managesent 
support  use  of  the  system  (7:10). 

Baker  and  Freeland,  1975.  The  authors  presented  an  assessment 
of  the  current  literature  that  dealt  with  "quantitative  models  of  the 
R  and  D  project  selection  and  resource  allocation  decision."  The 
models  reviewed  were  divided  into  two  categories  ' —  benefit  measurement 
and  resource  allocation  (3:1164). 


Two  main  findings  resulted  from  this  research.  The  first  finding 
was  the  contention  that  more  empirical  research  was  needed  in  under¬ 
standing  the  R&D  environment  as  well  as  the  "behavior  process  by  which 
decision  and  information  systems  become  adopted  and  implemented" 
(3:1172). 

The  second  finding  was  that  decision  models  were  giving  way 
to  decision  information  systems.  Two  reasons  were  given  for  this 
phenomena.  First,  decision  models  do  not  encompass  all  the  relevant 
factors  which  are  used  in  R&D  decisions,  forcing  managers  to 
consistently  readjust  their  resource  allocations.  Second,  moat 
criteria  are  not  easily  quantified,  thereby  requiring  they  be  handled 
by  more  qualitative  methods  (3:1173). 

A  Selected  Survey  of  R&D  Project  Selection  Model  Research 

Asher ,  1962.  A  linear  programming  model  was  developed  for  the 
purpose  of  allocating  a  scarce  resource  in  the  pharmaceutical  industry. 
This  resource  was  professional  swnpower,  and  it  had  to  be  allocated 
among  many  alternative  research  projects  (2:154). 

Other  constraints  considered  were:  economic  value,  probability 
of  success,  manhours  required  to  test  or  screen  a  project,  manhour 
availability,  cost  per  manhour  and  raw  material  availability.  All 
these  constraints  were  integrated  in  the  formulation  of  a  linear 
programming  model  (2:154). 

The  solution  obtained  indicated  "the  optimum  allocation  of 
professional  manpower  over  the  most  attractive  projects  to  maximise  the 
return  to  the  corporation"  (2:154). 


Watters,  1967,  The  author  examined  the  problem  of  R&D  project 
portfolio  selection  in  organizations.  He  took  the  perspective  of 
"investing"  projects  in  an  economic  sense  and  sought  to  construct  a 
method  that  balanced  both  profitability  and  investment  risk  (19:2-3). 

The  method  combined  aspects  of  utility  theory,  probability  theory, 
and  mathematical  programming.  This  model  is  a  systematic  technique 
that  can  be  used  in  considering  factors  which  other  models  do  not 
address  (19:135). 


Specifically,  a  method  was  developed  for  solving  R&D  project 
selection  problems  in  which  (1)  cash  flows  are  not  known  with 
certainty,  (2)  some  or  all  of  the  investment  opportunities 
are  interrelated,  (3)  limited  funds  in  multiple  fiscal 
periods  necessitates  the  imposition  of  multiperiod  proba¬ 
bilistic  budget  constraints,  and  (4)  the  suitability  of 
undertaking  a  given  portfolio  of  investment  opportunities 
depends  upon  both  profitability  and  wish  considerations. 
[19:135] 


Moore  and  Baker,  1969.  The  authors  performed  a  computational 
analysis  of  scoring  models  and  their  application  to  R&D  project 
selection.  They  stated  that  the  major  weakness  of  these  types  of 
models  is  their  arbitrary  construction  and  the  inability  of  model 
builders  to  deal  with  the  impact  of  certain  structural'  considerations 
on  project  scores  (15:B-212). 

Two  principal  research  questions  were  investigated: 


1.  Is  it  possible  to  construct  a  scoring  model  such  that 
its  performance  is  consistent  with  other  models  having 
economic  and  constrained  optimization  structures? 

2.  If  such  consistency  can  be  achieved,  what  properties 
of  the  scoring  model's  structure  are  responsible  for  the 
consistency?  [15:B-214] 
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The  methodology  used  involved  an  analytical  approach  as  opposed 
to  a  theoretical  approach.  A  tactical  simulator  was  generated  that 
compared  the  behavior  of  a  scoring  model  against  two  other  models  that 
represented  economic  and  constrained  optimization  classes  (15:B-2l2). 

Moore  and  Baker  made  some  observations  about  scoring  models. 
Scoring  models  were  stronger  tools  than  they  had  originally  thought. 

Its  strengths  lay  in  the  ability  of  the  method  to  process  economic, 
historical  and  other  environmental  types  of  data.  However,  further 
studies  of  projects  operating  in  a  real  R&D  setting  were  needed  to 
adequately  evaluate  performance  ( 15:B-230-231) . 

The  authors  state  that  their  research  supports  the  conjecture  that 
scoring  models  may  be  used  throughout  the  life  of  an  R&D  project  or 
they  can  be  used  for  various  evaluations.  They  caution  that  additional 
testing  with  actual  data  must  be  done  before  either  approach  is  adopted 
(15:8-231). 

Gear,  Lockett  and  Pearson,  1971.  The  researchers  concerned 
themselves  with  an  analysis  of  a  selection  of  R&D  portfolio  selection 
models.  The  research  methods  used  were  a  literature  review  to  gather 
inputs  (models)  and  an  analysis  of  such  models  to  determine  strengths 
and  weaknesses  (10:66). 

The  models  are  classified  according  to  whether  they  are 
based  on  linear,  integer,  chance  constrained  or  dynamic 
programming.  .  .  .  The  evaluation  is  in  terms  of  data 
requirements;  built-in  assumptions;  ease  of  computation; 
usefulness  of  output;  versatility  of  application.  [10:66] 

The  authors  confirm,  after  brief  evaluations  of  each  model,  that  it  is 
quite  difficult  to  pick  an  appropriate  model.  Further  testing  using 


real  R&D  field  data  is  required  for  just  about  all  of  the  models  that 
the  authors  evaluated  (10:75). 

Souder ,  1975.  The  author  used  an  organizational  behavior 
technique,  called  an  impact  method,  to  attempt  an  organizational 
consensus  in  specifying  R&D  project  selection  criteria.  The  experiment 
encompassed  four  different  organizations,  known  only  as  companies  A  and 
B  with  divisions  X  and  Y  (18:669). 

The  method  consists  of  the  repeated  use  of  a  paired 
comparison  instrument,  with  group  discussions  and  member 
interactions.  ...  It  was  concluded  that  the  impact 
method  may  be  generally  useful  for  disclosing  hidden  value 
conflicts.  However,  the  achievement  of  high  levels  of 
shared  values  and  decision  consensus  may  be  inhibited 
where  either  a  strong  central  leader  is  lacking  or  where 
individuals  are  unclear  about  the  nature  of  the  larger 
goals  of  the  organization.  [18:669] 

Souder  believes  that  this  procedure  is  useful  in  planning  and 
policy  formulations  in  addition  to  R&D  planning.  However,  consensus 
varies  on  the  willingness  of  participants  to  allow  open  conflict.  When 
open  conflict  is  repressed,  cliques  of  dissatisfaction  often  arise, 
seriously  impinging  the  consensus  process  (18:680). 

Aaker  and  Tyebee ,  1978.  The  authors  constructed  a  model  that 
dealt  with  the  selection  of  interdependent  R&D  projects.  The  model 
covers  three  areas  of  interdependence:  resource  utilization,  technical 
overlap,  and  interaction  with  respect  to  value  contribution.  The  model 
is  also  structured  to  assist  people  with  diverse  backgrounds  in  an 
organization  to  convey  their  inputs  into  the  R&E  planning  process 


Typically,  the  R&D  group  would  best  be  able  to  estimate  the 
project  costs,  including  resources  overlap,  generate  the 
various  probability  inputs  required,  and  identify  technically 
dependent  projects.  The  profit  implications  of  profit 
outcomes  would  require  sales  forecasts  by  several  marketing 
groups  and  cost  estimates  by  production  managers.  Finally, 
global  budget  constraints,  internal  rate-of-return 
constraints,  and  the  identification  of  the  long  range 
strategic  value  of  financial  managers  and  top  management. 

The  model  can  thus  be  viewed  as  a  vehicle  by  which  these 
varied  organizational  groups  will  comounicate  and  interact 
during  the  R&D  funding  decision-making  process.  [1:36] 


Brooks ,  1979.  The  author  formulated  a  descriptive  method  called 
"policy  capturing"  in  an  attempt  to  model  the  decision  making  process 
that  Air  Force  laboratory  managers  used  when  making  R&D  project 
selection  decisions.  A  survey  of  laboratory  managers  was  conducted 
and  the  respondents  were  categorized  according  to  management  level, 
division,  and  type  of  laboratory  project.  Models  were  constructed  to 
determine  the  extent,  if  any,  that  a  consensus  on  R&D  project  selection 
existed  among  them  (6:viii). 

The  survey  instrument  incorporated  six  predictive  factors  that 
Air  Force  laboratory  personnel  seemed  to  use  the  most.  This  deter¬ 
mination  was  the  result  of  previous  work  by  Air  Force  Institute  of 
Technology  (AFIT)  faculty.  The  six  factors  were:  1)  cost-benefit 
ratio,  2)  technical  merit,  3)  resource  availability,  4)  likelihood 
of  success,  5)  time  period,  and  6)  Air  Force  need  (6:51-52). 

Captain  Brooks  concluded  that  his  study  supported  policy  capturing 
as  a  technique.  His  major  observation  was  that  a  consensus  in  R&D 
decision  making  did  not  exist  between  different  divisions,  nor  was 
their  much  consensus  in  this  area  within  divisions.  In  addition, 
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managers  often  did  not  use  a  decision  making  process  in  the  same 
manner  as  they  had  originally  perceived  that  method  (6:89). 

Captain  Brooks'  work  is  particularly  notable  in  that  it  is  one 
of  the  few  attempts  to  construct  a  descriptive,  as  opposed  to  a 
prescriptive,  model. 

Chiu  and  Gear,  1979.  The  authors  present  an  application  of 
stochastic  integer  programming,  formulated  to  a  portfolio  of  projects. 
Each  of  the  projects  was  planned  with  the  aid  of  a  decision  tree 
structure.  Subsequent  studies  were  conducted  after  a  year's  duration 
to  assess  performance  of  the  model  in  practice  (8:2). 

Chiu  and  Gear  concluded  that  the  decision  tree  structure  of 
the  model  adequately  projected  the  paths  that  the  research  projects 
actually  took.  As  a  minimum,  the  model  provided  a  convenient  starting 
point  for  interactive  mode  operation.  On  the  negative  side,  the  model 
did  not  adequately  deal  with  the  interdependencies  of  the  projects 
(8:5-6). 

Golabi ,  Kirkwood ,  and  Sicherman,  1981.  The  authors  described  a 
procedure  for  selecting  a  portfolio  of  R&D  solar  energy  projects.  The 
method  made  use  of  multiattribute  preference  theory  and  was  used  by 
the  U.S.  Department  of  Energy  (11:174). 


The  technical  quality  of  each  proposed  applications 
experiment  was  summarized  through  the  use  of  multiple 
evaluation  measures,  or  attributes.  These  were  combined 
into  a  single  index  of  the  overall  technical  quality  of 
an  experiment  through  the  use  of  a  multiattribute  utility 
function.  Recently  derived  results  in  measurable  value 
theory  were  applied  to  derive  an  index  of  the  overall 
technical  quality  of  a  portfolio  of  experiments.  Budgeting 
and  programmatic  issues  were  handled  through  the  use  of 
constraints.  This  approach  allowed  the  portfolio  selection 
problem  to  be  formulated  as  an  integer  linear  program. 
[11:174] 
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The  authors  report  that  the  procedure  decreased  dramatically  the 
time  it  took  to  evaluate  R&D  project  proposals.  However,  implemen¬ 
tation  requires  proper  management  techniques  and  a  heavy  degree  of 
computer  support  (11:188). 

A  Review  of  Some  Commonly  Recurring  Selection  Mode 1 s 

The  literature  in  this  area  is  rich  in  documenting  the 
applications  and  innovations  that  have  evolved  in  the  field  over 
the  past  quarter  century.  The  selection  methods  here  differ  from 
the  previously  described  models  in  that  they  can  be  used  either 
independently  or  as  components  or  subelements  of  models.  Some  of 
the  more  common  techniques  will  be  discussed  here.  They  include: 

1)  checklist  and  profile  charts,  2)  scoring  models,  3)  cost/benefit 
ratios,  4)  decision  trees,  3)  linear  programming,  6)  goal  programming, 
7)  dynamic  programming,  8)  chance  constrained  programming,  and 
9)  multiple  objective  techniques. 

Checklist  and  Profile  Charts.  This  method  is  regarded  by  many 
in  the  field  as  the  simplest  of  the  techniques.  Its  use  involves 
completing  a  Checklist  for  the  project  under  consideration  (13:16). 

The  method  works  in  the  following  manner: 

Criteria  are  listed  which  are  believed  to  be  important 
factors  in  determining  the  eventual  success  or  failure  of 
the  R&D  effort  and  the  ultimate  product.  Each  candidate 
project  is  then  subjectively  rated  on  the  basis  of  each 
criteria  listed.  The  opinions  of  several  individuals  could 
be  summarized  in  checklist  by  averaging  their  opinions. 

[18:16] 
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Some  of  the  advantages  of  checklists  are:  1)  they  are  simple  and 
easy  to  use,  2)  criteria  are  easily  matched  with  available  information, 
3  they  accommodate  information  that  does  not  fit  well  into  other  more 
structured  models  (noneconomic  factors,  social  impacts,  environmental 
concerns),  and  4)  they  identify  project  weaknesses  quickly  via  a 
criteria  rating  system  (13:16). 

Among  the  disadvantages  often  cited  are:  1)  complex  problems 
are  overlooked,  2)  complex  interrelationships  with  other  projects  are 
overlooked,  3) individual  factors  are  not  prioritized,  and  4)  results 
could  be  affected  by  inaccurate  information  from  respondents  (13:16). 

Scoring  Models.  Scoring  techniques  are  the  next  level  of 
sophistication  up  from  checklist  and  profile  charts.  "Scoring  models 
compute  an  overall  project  score  based  on  ratings  assigned  to  each 
project  for  each  relevant  decision  criteria  and  are  designed  to  operate 
with  the  subjective  input  data  which  exists  as  the  research  stages  of 
the  project  life"  (15:213). 

Byron  Jackson  relates  some  of  the  advantages  and  disadvantages  of 
this  technique: 


Scoring  models  retain  the  advantages  of  checklist  and  profile 
charts  in  terms  of  their  ability  to  consider  a  wide  range  of 
economic  as  well  as  non-economic  criteria.  In  addition, 
scoring  models  make  it  possible  to  provide  a  single  number 
evaluation  for  each  project  and  they  areeasily  modified  to 
meet  conditions  (Dean  &  Nishry,  1964).  The  cost  of  this 
improvement  is  a  significant  increase  in  information 
requirements.  The  principle  shortcomings  of  the  approach 
are  that  the  project  is  dimensionless,  which  limits  its  use 
to  rank  order  comparisons;  and  the  model  development  is 
nonformal,  which  makes  it  difficult  in  some  situations  to 
justify  its  use  as  opposed  to  economic  or  optimization 
models.  [13:18] 
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Cost/Benefit  Ratios.  This  technique  attempts  to  balance  project 
risks  and  project  costs  to  aid  the  R&D  decision  maker. 

The  techniques  associated  with  cost-benefit  analysis  allow 
a  decision  maker  to  choose  between  alternative  research 
projects  by  systematically  evaluating  the  benefits  and  costs 
associated  with  each  project  into  monetary  equivalents.  The 
project  that  provides  the  greatest  net  monetary  benefit  to 
the  organization  is  then  selected  for  further  funding  and 
development.  Since  the  time  period  during  which  substantial 
costs  are  involved  frequently  varies  from  project  to  project, 
as  does  the  time  period  during  which  benefits  are  expected  to 
be  received,  the  time  value  of  money  is  a  critical  factor  in 
the  cost-benefit  approach  to  evaluating  research  projects. 

The  relative  risk  associated  with  each  project  must  also  be 
considered  in  determining  which  of  a  number  of  projects 
merits  the  resource  investment  required  for  development. 

[5:39] 

The  major  benefit  of  using  a  cost/benefit  ratio  is  that  it  forces 
decision  makers  to  quantify  their  evaluations  of  a  project.  Thi9 
quantification  process  forces  decision  makers  to  evaluate  their 
projects  with  a  clearer  perspective.  In  addition,  the  method  provides 
a  single  index  to  which  other  projects  may  be  compared,  thus 
simplifying  the  decision  process  (13:18). 

The  major  disadvantage  of  using  this  method  is  that  not  all 
non-economic  factors  can  be  translated  into  dollar  values.  An  example 
of  this  is  the  failure  of  the  technique  to  handle  resource  constraints. 
Also,  results  are  generally  sensitive  to  the  escalator  factor  used  in 
net  present  value  (NPV)  calculations. 

Decision  Trees.  Decision  trees  are  a  useful  technique  for 
graphically  displaying  possible  outcomes  and  their  probabilities  of 
occurrence  (14:810).  Decision  trees  address  a  major  problem  of 


project  selection;  the  interrelated  nature  of  many  research  projects 
(13:18-19). 

Decision  trees  have  two  strong  points.  First,  "they  focus 
attention  on  the  individual  subproject  which  make  up  a  complete  R&D 
project."  Second,  they  "provide  a  more  accurate  description  of  the 
R&D  decision  process  and  offer  a  better  basis  for  making  decisions 
than  other  methods"  (13:20). 

This  technique  has  two  fundamental  shortcomings.  It  requires  a 
great  deal  of  information,  some  of  which  is  not  always  available,  and 
it  does  not  adequately  deal  with  resource  constraints  (13:20). 

Linear  Programming.  A  linear  programming  model  incorporates 
linear  model  solution  techniques.  "A  linear  model  is  one  in  which  all 
of  the  functional  relationships  between  the  variables  in  the  models 
are  expressed  in  linear  terms"  (14:13). 

Linear  programming  is  used  by  commercial  managers  as  "the  most 
fundamental  quantitative  tool  of  R&D  projects  while  recognizing  limits 
on  the  available  resources  to  carry  out  the  projects"  (12:21). 

Linear  programming  formulation  generally  consists  of  three  parts. 
First,  the  model  requires  an  objective  function  which  the  user  attempts 
to  maximize  or  minimize.  Second,  the  model  contains  a  set  of  linear 
constraints,  which  represent  resource  constraints  of  one  sort  or 
another.  Finally,  nonnegativity  restrictions  are  defined  for  resources 
that  cannot  be  negative  (14:73-74). 

Some  of  the  reservations  R&D  managers  have  expressed  about  linear 
programming  are:  1)  it  has  large  information  requirements,  2)  it  does 
not  handle  uncertainties  in  the  R&D  environment  very  well,  3)  it  fails 


to  handle  project  interdependencies,  and  4)  fractions  of  projects  carry 
over  into  the  solution,  distorting  the  output  (12:22). 

Goal  Programming.  Goal  programming  is  a  relatively  new  technique. 
In  many  respects,  it  is  a  modification  of  linear  programming  that  deals 
with  multiple  objective  goals.  Markland  describes  the  technique  as 
follows: 

Goal  programming  .  .  .  allows  the  decision  maker  the 
opportunity  to  include  in  the  problem  formulation  multiple 
goals  or  objectives.  Goal  programming  greatly  enhances 
the  flexibility  of  linear  programming  as  it  allows  the 
inclusion  of  conflicting  objectives  while  still  yielding 
a  solution  that  is  optimal  with  respect  to  the  decision 
maker's  specification  of  goal  priorities.  The  use  of  goal 
programming  thus  reflects  a  philosophy  of  trying  to  obtain 
an  optimal  compromise  solution  to  a  set  of  conflicting 
objectives.  Goal  programming  has  been  applied  to  numerous 
multiobjective  modeling  situations,  including  linear  or 
nonlinear  functions  and  constraints,  and  both  continuous 
and  discrete  variables.  [14:254J 

Nussbaum  describes  the  formulation  of  a  goal  programming  problem 
(17:30): 

1.  Isolate  the  decision  variable. 

2.  Determine  the  goals/objectives  of  the  decision  maker. 

3.  Place  the  goals  into  priority  levels. 

4.  Link  decision  variables  and  goals  in  a  way  similar 
to  that  found  in  a  linear  programming  setup. 

The  effectiveness  of  goal  programming  lies  in  its  ability  to 
minimize  deviations  from  the  goals  (17:30).  Herein  also  lies  its 
biggest  defect.  The  results  it  yields  are  dependent  upon  the  way  those 
goals  are  articulated.  It  also  has  the  same  interrelatedness  problem 
of  linear  programming. 


Dynamic  Programming.  Dynamic  programming  incorporates  dynamic 
models.  A  dynamic  model  deals  with  multiple  tine  periods  in  selecting 
the  optimal  project  alternative.  A  series  of  interrelated  decisions 
are  made  that  cover  several  time  periods  (14:13). 

Dynamic  programming  is  really  a  general  type  of  problem 
solving  procedure  that  can  be  applied  to  sequential 
decision-making  situations.  The  dynamic  programming  model, 
or  set  of  equations,  that  is  formulated  must  be  developed 
uniquely  for  each  problem  solving  situation.  [14:556] 

The  chief  benefit  of  dynamic  programming  is  the  way  it  handles  the 
probability  of  technical  success.  This  is  due  to  the  ability  of  this 
technique  to  formulate  a  nonlinear  relationship  between  its  inputs  and 
the  expected  values  of  its  projects. 

This  method  is  also  effective  in  resource  allocation.  "If 
resource  expenditures  undergo  a  declining  rate  of  contribution  as  more 
are  expended  in  any  one  period,  the  resources  are  reallocated  to  other 
periods  so  as  to  maximize  their  overall  contribution  to  project  success 
(12:24). 

Two  problems  are  identified  with  this  method.  To  use  dynamic 
programming  it  is  necessary  to  determine  probabilities  of  technical 
success  as  a  function  of  past  and  current  research  spending.  This 
is  often  a  complex  task  in  and  of  itself.  Also,  only  one  resource 
constraint  can  be  considered  at  any  one  time  (12:24). 

Chance  Constrained  Programming.  This  method  treats  resources 
as  random  variables  rather  than  as  constant  parameters.  It  is  used 
primarily  for  assessing  R&D  project  portfolios  as  a  group,  after  one  of 
the  member  projects  of  that  portfolio  has  experienced  a  significant 


breakthrough  where  that  project  may  require  a  sudden  surge  of 
additional  resources  (12:24). 


The  second  constraint  limits  research  activities  at 
individual  institutions.  This  constraint  is  a  chance 
constraint  which  says  that  the  probability  that  the 
institution  has  sufficient  resources  to  carry  out  the 
research  activities  it  has  undertaken  is  at  least  equal 
to  minimum  values.  [12:24] 


This  model  has  a  number  of  disadvantages.  Foremost  among  them 
is  that  this  technique  does  not  lend  itself  to  easy  mathematical 
resolution.  In  addition,  models  of  this  nature  have  extensive  data 
requirements  (12:24). 

Multiple  Objectives,  this  technique  involves  the  use  of  objective 
functions  which  incorporate  multiple  objectives.  This  method  can  be 
used  under  conditions  of  either  certainty  or  uncertainty.  Many 
applications  of  multiattribute  theory  have  found  their  way  into  this 
method  (12:25). 


The  contribution  of  multiattribute  decision  theory 
is  twofold.  The  theory  recognizes  the  problems  of  project 
selection  where  different  scales  are  appropriate  for 
measuring  the  multiple  objectives  of  the  decision  maker, 
and  the  decision  maker  is  not  indifferent  to  the  uncertainty 
surrounding  the  outcome  of  the  R&D  project.  [12:26] 

There  are  some  serious  drawbacks  to  this  method.  First,  decision 
makers  must  provide  great  amounts  of  data.  Second,  the  method  is  very 
much  dependent  on  the  decision  maker  being  familiar  with  all  attribute 
of  his  project.  This  is  not  always  the  case.  Finally,  this  method 
often  requires  the  use  of  nonlinear  programming  techniques,  many  of 
which  are  quite  difficult  to  solve  (12:26). 


III.  Methodolo, 


This  chapter  reviews  the  methodology  that  was  used  to  collect 
and  analyze  the  data  for  this  effort.  The  first  section  of  the 
chapter  covers  the  scope  of  the  study  includes  the  laboratories  that 
participated  in  the  study,  the  sample  size  of  each  laboratory,  and  the 
data  gathering  method  used.  The  second  section  details  the  questions 
used  in  the  study  and  the  rationale  behind  their  use.  The  third 
section  describes  the  data  analysis  techniques  used.  The  final 
section  deals  with  the  limitations  of  this  research  effort. 

Scope 

The  data  collection  method  used  in  this  study  consisted  of  a 
combination  of  personal  and  telephone  interviews.  The  interviews 
consisted  of  twenty-two  questions.  A  copy  of  the  interview  questions 
is  provided  in  Appendix  A. 

The  primary  reason  for  the  selection  of  the  interview  method  was 
the  exploratory  nature  of  the  research.  No  previous  data  bases  existed 
that  could  have  provided  a  suitable  structure  around  which  to  build  an 
adequate  questionnaire.  Another  reason  for  the  use  of  the  interview 
method  was  that  it  provided  for  the  flexibility  of  responses  from 
laboratory  managers  being  questioned.  They  could  verbally  add 
impromptu  comments  which  they  normally  would  not  have  written  down. 
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The  four  laboratories  chosen  for  this  research  were  those  that 
constitute  the  Air  Force  Wright  Aeronautical  Laboratories  (AFWAL). 

They  are:  the  USAF  Avionics  Laboratory,  the  USAF  Aero-Propulsion 
Laboratory,  the  USAF  Flight  Dynamics  Laboratory,  and  the  USAF  Materials 
Laboratory.  All  four  elements  of  the  AFWAL  are  located  at  Wright- 
Pat  terson  AFB,  Ohio.  The  four  laboratories  of  AFWAL  were  chosen  for 
this  research  due  to  the  volume  of  in-house  research  projects  that  are 
conducted  there  and  also  because  of  their  close  geographical  proximity 
to  AFIT. 

A  sample  of  ten  people  from  each  laboratory  was  interviewed  for 
a  total  of  forty  interviews.  The  individual  interviews  were  selected 
in  the  following  manner.  First,  a  senior  laboratory  official  in  each 
laboratory  was  contacted  to  determine  which  managers  would  be  most 
suitable  for  interviewing.  This  senior  official  was  associated  with 
a  laboratory's  long  range  planning  (XR)  office.  Each  of  the  managers 
recommended  from  this  initial  meeting  was  contacted  and  an  interview 
request  was  made.  If  the  manager  responded  positively  an  interview 
was  conducted.  Each  of  these  managers  in  turn  recousended  additional 
managers  who  would  consent  to  be  interviewed. 

The  sole  criteria  of  participation  was  that  the  individual  be 
involved  in  selection  of  an  in-house  research  project.  However,  it  was 
not  required  that  the  participant  be  the  sole  selector  of  the  research 
project. 

The  interviews  used  a  combination  of  face-to-face  and  telephone 
methods.  While  the  former  was  the  most  desired  method,  because  of 
its  flexibility,  time  limitations  made  use  of  the  telephone  method  a 
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necessity.  Limitations  of  these  methods  will  be  discussed  later  in 
this  chapter. 

The  Interview  Format 

The  interview  format  was  designed  to  answer  the  ten  research 
objectives  that  were  stated  in  Chapter  I.  This  section  explains 
the  rationale  behind  the  interview  questions  found  in  Appendix  A. 

Questions  1  through  4  provided  demographic  data  in  the  form  of 
the  laboratory  where  the  individual  worked,  whether  he/she  was  in  the 
military  or  in  the  civil  service,  his/her  rank  or  grade  and  years 
experience  working  in  a  laboratory  environment. 

Question  5  determined  the  respondent's  highest  level  of  education. 

Question  6  determined  the  respondent's  management  level  in  the 
laboratory  organization. 

Question  7  sought  the  participant's  major  field  of  study. 

Question  8  sought  an  individual's  operations  research  background 
to  be  used  to  answer  the  third  research  objective. 

Questions  9  and  10  were  the  inputs  to  answer  the  sixth  research 
objective,  which  was  to  determine  a  respondent's  formal  laboratory 
management  training. 

Question  11  was  combined  with  question  15  to  answer  the  seventh 
research  objective  —  how  the  size  of  a  manager's  budget  would  impact 
his  inclination  to  use  a  formal  decision-making  method. 

Questions  12  and  13  were  combined  with  question  15  to  answer  the 
eighth  research  objective  —  how  the  selection  and/or  assignment  volume 
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of  an  Individual  influences  that  person's  prepensity  to  use  a  formal 
decision  method. 


Question  14  was  the  input  to  answer  the  first  research  objective, 
which  involved  gathering  data  concerning  project  selection  factors  used 
in  AFWAL. 

Question  15  was  the  input  to  answer  the  second  research  objective, 
which  was  to  ascertain  the  extent  AFWAL  researchers  and  managers  used 
formal  decision  methods  in  selecting  research  projects. 

Question  16  was  the  input  to  answer  the  third  research  objective, 
which  focused  on  decision-making  techniques  now  being  used  at  AFWAL. 

Question  17  was  combined  with  question  20  to  answer  the  ninth 
research  objective,  which  sought  to  determine  if  a  project's  dollar 
value  directly  influenced  the  formal  decision  method  used. 

Question  18  was  the  input  to  answer  the  fourth  research  objective, 
which  focused  on  decision  methods  respondents  were  aware  of  but  did  not 
use. 

Question  19  was  the  input  to  answer  the  fifth  research  objective, 
which  was  an  attempt  to  gauge  a  participant's  desire  to  change  project 
selection  methods. 

Question  21  was  the  input  to  answer  the  tenth  research  objective, 
which  dealt  with  project  portfolio  selection. 

Analysis 

The  data  collected  from  the  interviews  was  recorded  and  analyzed. 
The  analysis  techniques  used  were  frequency  response  and  crosstab 
correlation  of  two  or  more  variables.  These  techniques  were  utilized 


via  the  Statistical  Package  for  the  Social  Sciences  (SPSS),  a  computer 
based  statistics  package  (16).  Upon  completion  of  the  data  analysis, 
conclusions  were  drawn  and  recommendations  were  made. 

Limitations 

The  personal  interview  method  has  two  problems  associated  with 
it  —  bias  and  cost  (9:299).  Cost  was  not  a  problem  in  this  research, 
but  the  problem  of  bias  had  to  be  addressed.  Emory  defines  three 
components  of  bias:  sampling  error,  nonresponse  error,  and  response 
error  (9:299). 

Sampling  Error.  The  sampling  error  is  measured  in  terms  of 
its  validity.  Validity  has  two  components  —  accuracy  and  precision. 
Accuracy  is  "the  degree  to  which  bias  is  absent  from  the  sample.” 
Precision  is  defined  as  "precision  of  estimate"  (9:148). 

The  nature  of  the  research  for  this  project  was  of  an  exploratory 
nature.  No  known  studies  of  research  of  this  type  concerning  the  Air 
Force  laboratories  in  general  and  AFWAL  in  particular  were  available. 
Sampling  was  based  on  interviews  with  laboratory  managers  who  were 
involved  in  the  in-house  research  project  selection  process.  This  type 
of  sampling  is  justified  on  the  basis  of  the  following  argument  by 
Emory : 

[Such  samples]  are  appropriate  at  the  earliest  stages  of 
a  research  design,  when  one  is  first  attempting  to  develop 
hypotheses  and  procedures  for  measuring  them.  Then,  along 
with  reading  the  literature  and  discussing  ideas  with 
colleagues,  friends  and  relatives,  exploratory  date  gathering 
is  worthwhile.  Any  sort  of  sample  may  be  useful  when  very 
little  is  known.  Just  a  few  interviews  can  pinpoint  major 
problems  with  questions  and  dimensions  of  the  project  that 
the  research  may  have  ignored.  [9:179] 


Nonresponse  Error.  This  error  type  results  when  respondents  do 
not  wish  to  be  interviewed  or  when  respondents  cannot  be  found  (9:299). 
None  of  the  respondents  refused  to  be  interviewed  when  contacted. 
Establishing  initial  contact  with  prospective  respondents  was  the  chief 
problem  in  this  area. 

Respondent  Error.  This  error  bias  is  the  difference  between 
reported  and  actual  data  (9:301).  There  are  a  number  of  sources  for 
this  error  type:  1)  errors  in  processing  or  tabulating  data,  2)  failure 
of  a  respondent  to  provide  complete  and  accurate  information,  and 
3)  bias  caused  by  the  researcher  (9:301-302). 

Error  bias  in  processing  and  tabulating  data  was  ameliorated  by 
the  use  of  manual  and  computer  generated  statistical  methods.  A 
preplanned  questionnaire  was  used  for  all  the  interviews  to  limit 
interviewer  bias.  Respondent  bias,  however,  is  difficult  to  verify 
so  respondent  accuracy  will  be  assumed. 


IV.  Data  Analysis 


Introduction 

This  chapter  reflects  an  analysis  of  the  data  obtained  from 
the  forty  interviews.  The  data  is  summarized  and  presented  by  the 
following  topical  areas:  1)  demographic  data,  2)  research  project 
selection  factors,  3)  the  extent  of  formal  decision-making  method 
use  by  laboratory  managers,  4)  decision-making  techniques  used  by 
the  participants,  5)  decision-making  techniques  the  respondents  were 
aware  of,  6)  the  extent  that  the  sampled  individuals  desired  to  change 
their  selection  methods,  7)  participant's  formal  laboratory  training, 
8)  manager 'a  budget  size  vs  formal  decision  method  use,  9)  project 
selection/assignment  vs  formal  decision  method  use,  10)  selection 
method  propensity  vs  project  value,  and  11)  project  portfolio 
selections. 

Demographic  Data 

Each  of  the  four  laboratories  (Avionics,  Aero  Propulsion,  Flight 
Dynamics  and  Materials)  of  AFWAL  participated  in  the  study,  with  a 
sampling  of  ten  participants  from  each  lab  for  a  total  of  forty 
interviews.  Of  these  forty  participants,  four  were  active  duty 
military  and  thirty-six  were  civilian  government  employees. 

The  ranks  and  grades  of  the  participants  were  heavily  skewered 
towards  the  upper  levels  (Table  B.l).  The  breakout  of  the  military 
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respondents  was:  one  lieutenant  colonel,  one  major  and  two  first 
lieutenants.  Of  the  civil  service  employees  interviewed,  nearly 
three-fourths  (71%)  were  in  the  senior  grades  of  GS-13  through 
GS/GM-15.  Correspondingly,  nearly  two-thirds  (652)  had  ten  or 
more  years  experience  in  laboratory  work  (Table  B.2). 

The  research  sample  was  also  a  very  highly  educated  group  (Table 
B.3).  All  but  one  of  the  respondents  had  at  least  a  baccalaureate 
degree.  Over  two-thirds  (67.5%)  had  a  master's  and/or  Ph.d  degree. 
The  major  field  of  study  (Table  B.5)  was  engineering  (65%).  Science 
was  the  next  most  selected  field  (32.52).  One  participant  was  a  math 
major  and  no  one  had  a  non-technical  degree. 

While  the  pay  grades  of  the  participants  were  weighted  heavily 
towards  the  upper  grades,  the  management  levels  were  weighted  in  the 
opposite  direction  (Table  B.4).  Moat  of  the  participants  were  at  the 
researcher  level  (40%),  with  those  occupying  supervisory  positions  at 
the  next  management  level  coming  close  behind  (37.5%).  Eight  branch 
chiefs  and  one  directorate  head  rounded  out  the  sample. 

Research  Project  Selection  Factors 

The  overall  rating  for  the  order  of  importance  of  the  factors 
listed  was  determined  by  the  number  of  participants  who  indicated  a 
particular  factor's  level  of  importance  by  rating  it  on  a  scale  from 
one  (most  important)  to  nine  (least  important).  A  given  scale  level 
could  have  multiple  factors  (i.e.,  a  respondent  could  determine  that 
there  should  be  two  number  one  factors). 


The  overall  sample  ranking  of  factors  from  most  important  to  least 
important  are  as  follows  (Table  B.13): 

1.  Air  Force  Need 

2.  Technical  Merit 

3.  Resource  Availability 

4.  Likelihood  of  Success 

5.  Timeliness  of  Completion 

6.  Keep  Contractor  Honest 

7.  Educating  Personnel 

8.  Cost/Benefit  Ratio 

9.  Other  (unlisted)  Factors 

The  ranking  of  factors  by  laboratory  is  as  follows: 

Avionics  Laboratory  (Table  B.13-A. 1): 

1.  Technical  Merit 

2.  Air  Force  Need 

3.  Resource  Availability 
Educating  Personnel 

4.  Timeliness  of  Completion 

5.  Keep  Contractor  Honest 

6.  Cost/Benefit  Ratio 

7.  Likelihood  of  Success 

Aero  Propulsion  Laboratory  (Table  B.13-A.2): 

1.  Technical  Merit 

2.  Air  Force  Need 

3.  Cost/Benefit  Ratio 

4.  Likelihood  of  Success 

5.  Timeliness  of  Completion 

6.  Educating  Personnel 

7.  Keep  Contractor  Honest 

8.  Other 

9.  Resource  Availability 
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Flight  Dynamics  Laboratory  (Table  B.13-A.3): 


1.  Air  Force  Need 

2.  Cost/Benefit  Ratio 
Technical  Merit 

3.  Resource  Availability 

4.  Likelihood  of  Success 

5.  Timeliness  of  Completion 
Educating  Personnel 

6.  Keep  Contractor  Honest 

7.  Educating  Personnel 

8.  Cost/Benefit  Ratio 

9.  Other  (unlisted)  Factors 

Materials  Laboratory  (Table  B.13-A.4): 

1.  Air  Force  Need 

2.  Technical  Merit 

3.  Resource  Availability 
Likelihood  of  Success 

4.  Timeliness  of  Completion 

5.  Cost/Benefit  Ratio 

6.  Educating  Personnel 
Keep  Contractor  Honest 

The  ranking  of  factors  by  management  level  is  as  follows 

Researcher  (Table  B.13-B.1): 

1.  Air  Force  Need 

2.  Technical  Merit 

3.  Resource  Availability 

4.  Likelihood  of  Success 

5.  Keep  Contractor  Honest 

6.  Timeliness  of  Completion 

7.  Educating  Personnel 

8.  Cost/Benefit  Ratio 

9.  Other 
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Supervisor  (Table  B.13-B.2): 


1.  Air  Force  Need 

2.  Technical  Merit 

3.  Resource  Availability 
Educating  Personnel 

4.  Timeliness  of  Completion 

5.  Likelihood  of  Success 

6.  Keep  Contractor  Honest 

Branch  Chief  (Table  B.13-B.3): 

1.  Technical  Merit 
Air  Force  Need 

2.  Resource  Availability 

3.  Cost/Benefit  Ratio 
Keep  Contractor  Honest 

4.  Likelihood  of  Success 

5.  Timeliness  of  Completion 
Educating  Personnel 

Directorate  Head  (Table  B.13-B.4): 

1.  Air  Force  Need 

2.  Cost/Benefit  Ratio 

3.  Technical  Merit 

4.  Resource  Availability 

5.  Likelihood  of  Success 

6.  Timeliness  of  Completion 
Formal  Decision  Method  Used 

Of  the  forty  managers  surveyed,  only  twelve  (30Z)  indicated  they 
used  a  formal  decision-making  method  (Table  B.14).  In  a  breakdown  by 
laboratory  of  those  managers  who  responded  positively,  four  came  from 
the  Avionics  lab,  three  each  came  from  the  Flight  Dynamics  and  Materials 
labs,  and  two  came  from  the  Aero  Propulsion  lab  (Table  B.14-A). 


In  a  breakdown  by  management  level,  the  branch  chiefs  had  the 
greatest  percentage  of  positive  use  with  50  percent  (four  of  eight), 
followed  by  a  33.3  percent  (five  of  fifteen)  response  for  the  super¬ 


visors.  Of  the  sixteen  researchers  interviewed,  only  three  (19Z)  used 
a  formal  decision  method  and  the  one  directorate  head  interviewed  did 
not  use  a  formal  method  (Table  B.14-B). 

In  a  breakdown  by  experience,  there  did  not  appear  to  be  any 
discernible  patterns.  Many  of  the  more  senior  people  did  say,  however, 
that  they  tended  to  trust  their  own  instincts  in  selecting  projects 
rather  than  use  a  formal  decision  technique. 

Formal  Decision-Making  Techniques 

The  techniques  that  were  listed  and  their  frequencies  of  response 
are  as  follows:  checklist  and  profile  chart,  scoring  models,  decision 
trees  and  goal  programming  were  each  used  by  five  of  the  respondents. 
Cost/benefits  ratios  were  used  by  four  of  the  participants,  two 
individuals  used  multiple  objective  methods  and  one  respondent  used 
dynamic  programming  (Table  B.15-A).  Surprisingly  enough,  no  one  used 
linear  programming. 

Those  respondents  who  indicated  they  used  a  formal  decision  method 
were  much  more  likely  to  use  one  of  the  techniques  listed  than  those  in 
the  sample  who  did  not  use  any  formal  decision  method  (Table  B.15-B). 
There  did  not,  however,  appear  to  be  much  of  a  difference  between  those 
with  an  operations  research/management  science  background  and  those 
with  no  such  background  (Table  B.15-C).  On  the  contrary,  it  appears 


form  the  data  sample  that  those  with  no  such  background  were  slightly 
more  likely  to  use  one  of  the  listed  techniques. 

Decision-Making  Technique  Awareness 

The  decision-making  techniques  the  respondents  were  aware  of  are 
listed  from  most  aware  to  least  aware  as  follows  (Table  B.17): 

1.  Cost/Benefit  Ratio 
Decision  Trees 

2.  Checklist  and  Profile  Chart 
Linear  Programming 

3.  Scoring  Models 

4.  Goal  Programming 

5.  Multiple  Objective  Programming 

6.  Dynamic  Programming 

7.  Chance  Constrained  Programming 

8.  Other  Techniques  (not  specified) 

In  a  breakdown  by  laboratory  (Table  B.17-A)  the  Avionics  and 
Flight  Dynamics  laboratories  appeared  to  be  more  aware  of  the  listed 
techniques  than  the  Aero  Propulsion  and  Materials  laboratories.  The 
respondents  from  the  Aero  Propulsion  lab  seemed  to  be  slightly  more 
aware  of  the  techniques  than  the  Materials  lab. 

An  analysis  of  the  managerial  levels  (Table  B.17-B)  reveals  that 
supervisors  appear  to  be  the  most  aware  of  these  techniques.  The 
researchers  and  branch  chiefs  were  about  even  on  a  percentage  basis. 
Since  there  was  only  one  directorate  head  interviewed,  an  analysis 
of  that  response  data  would  be  irrelevant. 


Respondent's  Desire  to  Change  Methods 

Of  the  forty  respondents  interviewed,  only  five  indicated  they 
were  considering  the  possibility  of  changing  their  selection  methods 
(Table  B.18).  Of  these  five,  four  were  from  the  Avionics  lab  and  one 
was  from  the  Flight  Dynamics  lab  (Table  B.18-A).  In  a  management 
breakdown,  two  of  the  five  participants  were  researchers  and  three 
were  supervisors  (Table  B.18-B). 

Participant 1 s  Formal  Laboratory  Management  Training 

This  area  was  divided  into  two  segments.  Those  respondents  who 
received  their  training  via  an  AFIT  Professional  Continuing  Education 
(PCE)  course  and  those  who  received  training  through  another  lab  course 
or  workshop. 

Twelve  of  the  forty  respondents  reported  taking  a  laboratory  PCE 
course  (Table  B.7).  Five  respondents  reported  taking  other  types  of 
lab  management  training  courses  (Table  B.8).  Well  over  half  (57. 5Z) 
had  no  such  formal  laboratory  management  training. 

Manager 's  Budget  vs  Formal  Decision  Methods 

The  formal  decision  methods  did  not  necessarily  have  to  be  one  of 
those  listed  in  the  survey.  Twelve  of  the  forty  respondents  indicated 
that  they  did  change  their  methods  in  response  to  the  size  of  their 
project  budget.  Of  those  who  responded  positively,  the  vast  majority 
had  budgets  which  exceeded  $1  million  (Table  B.10). 
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Project  Select ion/ Assignment  vs  Formal  Decision  Methods 

A  point  to  be  emphasized  here  is  that  project  selection  denotes 
in-house  research  project  selection  only,  while  project  assignment  may 
be  in-house  assignment  or  contract  project  assignment  under  a 
alboratory  manager's  control.  No  apparent  relationship  seemed  to 
reveal  itself  with  respect  to  project  selection.  Those  individuals 
selecting  two  projects  per  year  seemed  to  have  the  greatest  propensity 
to  use  a  formal  decision  method  (Table  B.ll).  For  projects  assigned 
(Table  B.12),  those  individuals  assigned  ten  or  more  projects  seemed 
inclined  towards  using  a  formal  decision  method. 

Selection  Method  vs  Project  Value 

Twelve  respondents  (30Z)  indicated  that  their  project  selection 
methods  varied  according  to  the  dollar  value  of  their  budgets  (Table 
B. 19) .  However,  none  of  the  participants  gave  exact  dollar  values 
where  one  method  transitioned  to  another.  They  would  simply  define  the 
transition  in  terms  of  going  from  a  "small  dollar"  project  to  a  "large 
dollar"  project. 


Project  Portfolio  Selection 

Nineteen  of  the  forty  participants  engaged  in  selection  of 
portfolios  of  research  projects  (Table  B.20).  Portfolio  project 
selection  involves  two  or  more  individual  research  projects  that  are 
used  to  obtain  a  single  broad  research  objective.  Of  the  techniques 
listed,  goal  programming  was  the  most  popular  choice,  followed  closely 
by  multiple  objective  methods.  Linear  programming  and  dynamic  program¬ 
ming  were  also  indicated  to  be  used  for  this  purpose  (Table  B.20-A). 


V.  Conclusions  and  Recommendations 


Introduction 

The  conclusions  and  recommendations  in  this  chapter  are  based  on 
the  data  analysis  of  the  forty  interviews  that  were  conducted  for  this 
research.  The  chapter  is  broken  down  into  the  following  sections: 

1)  research  project  selection  factors,  2)  formal  decision  method  use, 
3)  formal  decision-making  techniques,  4)  decision-making  technique 
awareness,  5)  selection  improvement,  6)  formal  laboratory  management 
training,  7)  budget  and  decision  method  relationship,  8)  project 
selections/assignments  per  year,  9)  decision  method  relationships, 
relationships  between  project  value  and  selection  method  use, 

10)  project  portfolio  selection,  and  11)  recommendations  for  further 
research. 

Research  Project  Selection  Factors 

The  top  three  factors  considered  to  be  the  most  important  by  most 
of  the  respondents  were  Air  Force  need,  technical  merit,  and  resource 
availability.  Air  Force  R&D  laboratory  managers  appear  to  match  their 
commercial  counterparts  in  meeting  organizational  goals  (Air  Force 
need),  professional  goals  (technical  merit),  and  matching  these 
ambitions  with  a  judicious  management  of  available  resources. 


Formal  Decision  Method 


The  vast  majority  (70%)  of  the  respondents  surveyed  did  not  use 
a  formal  decision-making  method.  Most  managers  surveyed  said  that 
projects  were  selected  via  a  group  consensus  of  researchers  or  by  a 
management  committee. 

Many  of  the  project  selection  decisions  are  made  at  the  branch 
chief  level.  It  then  may  be  no  accident  that  branch  chiefs  in  the 
survey  had  the  highest  positive  response  percentage  for  using  a  formal 
decision-making  method. 

Formal  Decision-Making  Techniques 

The  four  techniques  with  the  greatest  positive  response  were 
checklist  and  profile  charts,  scoring  models,  decision  trees  and  goal 
programming.  With  the  exception  of  goal  programming,  the  first  three 
methods  are  relatively  simple  to  learn  and  easy  to  use.  This  may  have 
been  a  factor  in  their  popularity.  The  response  to  goal  programming 
came  as  somewhat  of  a  surprise  in  that  it  is  one  of  the  more  complex 
decision-making  techniques.  Its  flexibility  in  incorporating  several 
objectives  could  be  the  reason  several  of  the  respondents  use  it. 

The  research  also  indicated  that  exposure  to  operations  research 
methods  did  not  appear  to  influence  the  use  of  such  techniques  for 
project  selection.  In  some  cases  exposure  produced  the  opposite 
effect.  Participants  felt  the  techniques  were  too  complex,  time 
consuming  and  generally  unwieldy  for  project  selection  use  at  their 


level. 


Decision-Making  Technique  Awareness 


The  techniques  the  respondents  were  aware  of  most  were  those 
i  methods  most  commonly  used  in  consaercial  laboratories.  It  was  also 

>  a  point  of  interest  that  linear  programming  ranked  ahead  of  goal 

\ 

programming  in  awareness;  whereas  it  was  ranked  below  it  in  actual  use. 
An  explanation  for  this  may  be  that  if  one  were  to  use  an  operations 

j  *  research  method,  goal  programming  would  be  more  flexible  than  linear 
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programming  due  to  the  ability  to  work  with  multiple  goal  objectives. 

The  significantly  greater  number  of  respondents  who  are  aware 

I 
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>  of  the  listed  techniques,  contrasted  with  the  relatively  low  response 
of  techniques  usage,  illustrates  two  points.  First,  since  project 
selection  in  most  of  AFtfAL  is  by  a  group  consensus  of  managers,  use 
of  these  methods  may  be  irrelevant.  Second,  the  techniques  themselves 
may  be  so  time  consuming  and  complex  (particularly  in  the  case  of  the 
operating  research  methods),  that  they  are  simply  impractical  to  use. 
These  two  findings  are  the  oust  profound  of  this  project.  They  tend  to 
indicate  that  most  of  the  decision-making  tools  are  impractical  because 
of  disharmony  with  the  existing  laboratory  management  style  and  the 
technical  weaknesses  of  the  methods. 

Respondent 1 s  Desire  to  Change  Methods 

Most  of  the  respondents  felt  their  selection  methods  worked  well 
and  this  was  reflected  in  their  answers  to  this  question.  Those  that 
responded  positively  to  this  question  did  not  specify  switching  to  a 
particular  method  or  technique.  The  respondents  merely  indicated  that 
their  selection  process  may  become  more/ less  formal  than  it  was  at 
present. 
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An  explanation  for  the  low  positive  response  to  this  question  may 
be  the  feeling  expressed  by  some  supervisors  and  researchers  that  they 
have  little  control  over  final  project  selection.  Final  decisions  on 
project  selection  are  often  reserved  for  the  branch  chief  level  or 
higher.  Branch  chiefs  felt  that  their  project  selection  decisions 
were  sound  ones  and  thus  had  little  reason  to  change. 

Formal  Laboratory  Management  Training 

Over  half  of  the  participants  in  this  study  did  not  take  a  PCE 
course  or  any  other  type  of  formal  laboratory  management  training 
course.  Of  those  who  did,  few  could  recall  if  project  selection  was 
an  area  covered  by  these  courses.  It  then  appears  that  attendance  at 
either  a  PCE  or  other  similar  lab  management  courses  had  little  impact 
on  a  manager's  selection  method. 

Manager 'a  Budget  vs  Formal  Decision  Methods 

About  one-quarter  of  the  respondents  indicated  that  a  relationship 
existed  between  these  two  factors.  Of  this  group,  the  feeling  was  that 
the  larger  the  budget  they  controlled  the  greater  was  their  propensity 
to  use  some  sort  of  formal  decision-making  process. 

There  is  an  obvious  reason  to  explain  this  trend.  Higher  budgets 
are  often  controlled  by  managers  in  upper  levels  of  the  organization. 
These  individuals  often  have  to  justify  their  budgets  to  still  higher 
management  levels.  Formal  decision-making  methods  are  an  accepted  tool 


for  such  a  team. 


Project  Select ion/ Assignment  vs  Formal  Decision  Method  Use 

No  relationship  appeared  between  the  number  of  projects  selected 
per  year  and  the  inclination  to  use  a  formal  decision  method.  However, 
there  did  seem  to  be  a  relationship  between  the  volume  of  projects 
assigned  per  year  and  the  use  of  some  sort  of  formalized  method.  This 
tendency  seemed  to  manifest  itself  in  those  managers  who  were  assigned 
the  responsibility  of  ten  or  more  projects.  The  reason  for  the  level 
of  frequency  response  in  this  category  may  be  that  upper  levels  of 
management  fell  into  this  bracket  most  often  and  are  under  greater 
scrutiny  than  are  managers  in  lower  levels  of  their  respective 
laboratory  organizations. 

Selection  Method  vs  Project  Value 

The  response  to  this  question  was  small  (30Z).  The  respondents 
how  did  vary  their  selection  methods  based  on  a  project's  dollar  value 
typically  stated  they  went  from  a  less  formal  method  to  a  method  that 
involved  a  greater  management  or  group  consensus.  They  also  could  not 
name  specific  transition  points  where  these  changes  occurred.  The  only 
conclusion  drawn  from  this  sample  is  that  a  perception  exists  that  a 
more  expensive  project  requires  a  greater  management  involvement  at  the 
time  of  selection. 

Project  Portfolio  Selection 

While  a  substantial  number  of  participants  engaged  in  portfolio 


selection,  most  who  did  treated  the  process  no  differently  than  single 
project  selection.  Correspondingly,  the  techniques  used  for  this 


activity  were  similar  to  those  used  for  selecting  a  single  research 
project.  Project  portfolio  selection  is  not  viewed  as  a  task  uniquely 
different  from  ordinary  single  project  selection  at  AFWAL. 

Recommendation  for  Further  Research 

This  exploratory  research  was  conducted  to  examine  how  research 
project  selection  is  conducted  at  AFWAL.  Further  research  should 
be  conducted  to  explore  some  of  the  aspects  of  the  research  project 
selection  process  in  the  Air  Force  laboratories  that  have  surfaced 
here. 

Some  of  those  areas  are:  the  use  of  group  consensus,  the  impact 
of  management  information  systems,  and  the  impact  a  decision  support 
system  may  have  on  research  project  selection  in  AFWAL  or  any  other 
Air  Force  laboratory.  In  addition,  this  study  could  be  repeated  using 
a  survey  to  obtain  a  broader  and  deeper  sample  from  each  of  the  four 
laboratories,  thus  permitting  the  formulation  of  more  conclusive 


observations. 


Appendix  A:  Interview  Questions 


NAME  (optional)  _ _ 

1 .  LABORATORY  _ 

2.  D  MILITARY  0  CIVIL  SERVICE 

3.  RANK/ GRADE  _ _ 

4.  YEARS  EXPERIENCE  _ 

5.  EDUCATIONAL  LEVEL: 

D  No  College 
D  AA 

D  Bachelor's 
D  Master's 
D  Doctorate 

6.  MANAGEMENT  LEVEL:  _ 


7.  MAJOR  FIELD  OF  STUDY: 

AA  _ 

Bachelors  _ 

Master's  _ 

Doctorate 


8.  Did  you  ever  take  an  operations  research  or  management  science 
course?  0  Yes  0  No 

9.  Did  you  ever  take  a  Professional  Continuing  Education  (PCE)  course 
in  laboratory  management?  Q  Yes  D  No 

If  Yes:  Name  of  Course:  _ 


Date  Taken: 


Did  this  course  cover  project  selection?  D  Yes  D  No 

10.  Have  you  ever  taken  any  other  type  of  lab  course  that  dealt  with 
the  subject  of  project  selection  (i.e.,  a  local  lab  course  or 
workshop)?  0  Yes  0  No 

11.  Size  of  budget  you're  responsible  for:  $ _ 

12.  How  many  projects  do  you  select  per  year?  _ 
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13.  How  many  projects  are  assigned  to  you?  _ 

14.  What  factors  do  you  consider  important  when  selecting  a  research 
project?  (Rank  by  Priority  1-9): 

_  Cost/Benefit  Ratio 

_  Technical  Merit 

_  Resource  Availability 

_  Education 

_  Keep  Contractor  Honest 

15.  Do  you  use  a  formal  decision-making  method  for  proposal  selection? 

D  Yes  D  No 

16.  If  Yes:  What  methods  do  you  use?  (Rank  by  Frequency  of 
Use  1-10): 

_  Checklist  and  Profile  Charts 

_  Scoring  Models 

_  Cost/Benefit  Ratios 

_  Decision  Trees 

_  Linear  Programming 

_  Goal  Programming 

_  Dynamic  Programming 

_  Chance  Constrained  Programming 

_  Multiple  Objective  Programming 

Other 


Likelihood  of  Success 
Timeliness  of  Completion 
Air  Force  Need 
Other 


17.  What  are  the  dollar  values  of  these  projects? 

D  Less  than  $100K  0  $300K  -  $1M 

D  $100K  -  $300K  0  Greater  than  $1M 

18.  What  project  selection  methods  are  you  aware  of? 

_  Checklist  and  Profile  Charts 

_  Scoring  Models 

_  Cost/Benefit  Ratios 

_  Decision  Trees 

_  Linear  Programming 

_  Goal  Programming 

_  Dynamic  Programming 

_  Chance  Constrained  Programming 

_  Multiple  Objective  Programming 

Other 


19.  Do  you  anticipate  changing  your  selection  methods  in  the  near 
future? 

D  Yes  0  No 

If  Yes,  please  describe  those  changes. 

20.  Does  the  selection  method  vary  according  to  the  dollar  value 
of  the  project? 

D  Yes  Q  No 

If  Yes,  what  method  would  you  choose  for  the  following 
project  dollar  values? 

Less  than  $100K  _ 

$ 100K  -  $300K  _ 

$300K  -  $1M  ' _ ~~~  _ 

Greater  than  $1M  _ 

21.  Do  you  select  portfolios  of  research  projects? 

0  Yes  D  No 

If  Yes,  what  methods  do  you  use? 

D  Linear  Programming 
D  Goal  Programming 
D  Multiple  Objective  Programming 
0  Dynamic  Programming 
0  Chance  Constrained  Programming 
Q  Other  (List): 


22.  Additional  Comments: 
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Appendix  B:  Summarized  Responses  to  Interview  Questions 


TABLE  B.l 

Rank/Grade 

Rank/Grade 

Responses 

1st  Lieutenants 

2 

Majors 

1 

Lieutenant  Colonels 

1 

TOTAL  MILITARY 

4 

GS-9 

1 

GS-11 

1 

GS-12 

8 

GS-13 

7 

GS-14 

8 

GM-15 

10 

TOTAL  CIVIL  SERVICE 

35 

1  Unknown 

- 

TABLE  B.2 

Years  of  Laboratory  Experience 


Years 

Experience 

Number  of 
Responses 

Less  than  2 

3 

2-4 

4 

5-10 

7 

11-20 

13 

21  -  30 

11 

More  than  30 

2 

TOTAL  RESPONSES 

40 

TABLE  B.3 

Educational  Level 

Level 

Frequency  of 

Achieved 

Response 

No  College 

1 

AA 

0 

Bachelor's  Degree 

12 

Master1 8  Degree 

14 

Doctorate  Degree 

13 

TOTAL 

40 

TABLE  B.4 


Management  Level 


Management 

Level 

Frequency  of 
Response 

Researcher 

16 

Supervisor 

15 

Branch  Chief 

8 

Directorate  Head 

1 

TOTAL 

40 

TABLE  B.5 

Major  Field  of  Study 

Academic 

Frequency  of 

Discipline 

Response 

Engineering 

26 

Science 

13 

Mathematics 

1 

Non-Technical 

0 

TOTAL 


40 


TABLE  B.6 


m 


Operations  Research/Management  Science  Background 


Response 

Frequency 

« 

Yes 

14 

• 

No 

26 

TABLE  B.7 

Respondents  with  PCE  Background 

Response 

Frequency 

Yes 

12 

No 

28 

1 

TABLE  B.8 

Respondents  Who  Have  Taken 

Other  Lab  Management  Courses 

1 

a 

Response 

Frequency 

> 

Yes 

5 

1  ’ 

! 

\ 

V 

No 

35 

t 

i 

t 
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TABLE  B. 11 


Relationship  Between  Number  of  Projects 
Selected  Per  Year  and  Use  of  an  FDM 


Use  of 
FDM 

1 

2 

Projects 
3  4 

Selected 

5  6 

Per 

7 

Year 

8 

9 

10 

Yes 

10 

5 

4 

0 

1 

0 

0 

4 

No 

1 

4 

2 

0 

2 

1 

0 

0 

0 

2 

TOTAL 

11 

9 

1 

5 

1 

1 

0 

H 

6 

TABLE 

B.  12 

Relationship  Between  Number  of  Projects 
Assigned  Per  Year  and  Use  of  an  FDM 

FDM 

1 

2 

Projects  Selected  Per 
3  4  5  6  7 

Year 

8 

9 

10 

Yes 

3 

0 

7 

5 

4 

2 

0 

0 

1 

6 

No 

0 

0 

1 

2 

0 

0 

0 

0 

0 

9 

TOTAL 

3 

0 

8 

7 

4 

2 

0 

0 

1 

15 
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TABLE  B. 13 


Selection  Factors  and  Their  Ranking  of  Importance 


Acronym: 


CBR 

TM 

RA 

LOS 

TOC 

EDPERS 

KCH 

AFN 

OTH 


Cos t  Benefit  Ratio  j 

Technical  Merit 

Resource  Availability  * 

Likelihood  of  Success 

Timeliness  of  Completion 

Educate  Personnel  (Lab  Project  Personnel 

Keep  Contractor  Honest 

Air  Force  Need  .  j 

Other 


Interpretation  of  Scale: 

1.  Most  Important 
9.  Least  Important 
0.  Not  Considered 


Table  B.13-A 


1 

2 

Overall  Frequency  of  Response 

3  4  5  6  7  8 

9 

10 

CBR 

5 

5 

8 

6 

3 

3 

3 

1 

6 

TM 

15 

17 

5 

1 

D 

D 

0 

0 

2 

RA 

3 

8 

13 

8 

2 

2 

D 

0 

0 

4 

LOS 

5 

4 

11 

6 

2 

3 

4 

0 

5 

TOC 

1 

3 

6 

7 

9 

3 

2 

0 

9 

EDPERS 

1 

4 

6 

3 

6 

3 

6 

2 

D 

9 

KCH 

D 

2 

3 

2 

4 

5 

8 

0 

11 

AFN 

22 

8 

2 

2 

1 

1 

0 

1 

0 

3 

OTH 

0 

0 

0 

0 

H 

0 

D 

0 

2 

38 
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Table  B.13-A.1 


1 


Table  B.13-A.3 


Laboratory  Frequency  of  Kesponses-Flight  Dynamics 


EDPERS 


1  1 

i  0 


Table  B.13-A.4 


Laboratory  Frequency  of  Responses  -  Materials 


CBR 

TM 

2 

3 

4 

4 

1  2 

3  0 

1 

0 

H 

1 

0 

0 

0 

0 

0 

0 

|  RA 

i 

0 

1 

4  4 

1 

0 

0 

0 

0 

0 

t 

LOS 

0 

1 

0  3 

2 

i 

0 

2 

0 

1 

TOC 

i 

0 

0 

1  1 

2 

3 

1 

0 

0 

2 

|  EDPERS 

0 

2 

1  0 

2 

1 

2 

0 

0 

2 

,  KCH 

0 

0 

0  1 

1 

1 

1 

2 

0 

3 

1 

|  AFN 

8 

2 

0  0 

D 

0 

0 

0 

0 

i 

|  OTH 

1 

0 

0 

0  0 

0 

0 

1 

0 

0 

10 

Table  B.13-B.1 


Table  B.13-B.3 


Management 
1  2 

Frequency  of  Responses  - 
3  4  5  6  7 

Branch  Chiefs 

8  9  10 

CBR 

0 

0 

1 

0 

2 

1 

2 

0 

0 

2 

TM 

4 

3 

1 

0 

0 

0 

0 

0 

0 

0 

RA 

0 

3 

3 

2 

0 

0 

0 

0 

0 

0 

LOS 

0 

1 

1 

4 

0 

1 

0 

0 

0 

1 

TOC 

0 

0 

0 

3 

2 

2 

0 

0 

0 

1 

EDPERS 

1 

2 

0 

0 

2 

0 

2 

1 

0 

0 

KCH 

0 

0 

1 

0 

0 

2 

1 

3 

0 

1 

AFN 

4 

0 

2 

1 

0 

0 

0 

0 

0 

1 

OTH 

0 

0 

0 

0 

0 

0 

0 

0 

0 

8 

Table 

B.13-B. 

4 

Management 
1  2 

Frequency  of  Responses  - 
3  4  5  6  7 

Director 
8  9 

Head 

10 

CBR 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

TM 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

RA 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

LOS 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

TOC 

0 

0 

0 

0 

0 

l 

0 

0 

0 

0 

EDPERS 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

KCH 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

AFN 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

OTH 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 
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TABLE  B. 14 


Formal  Decision-Making  Method  Use 


Overall  Response 


% 

Yes 

12 

No 

28 

I  Table  B.14-A 


Laboratory  Response 
Yes  No 

Avionics 

6 

4 

Aero  Propulsion 

8 

2 

Flight  Dynamics 

7 

3 

Materials 

7 

3 

Table 

B.14-B 

Management  Response 

Yes 

No 

Avionics 

13 

3 

Aero  Propulsion 

10 

5 

Flight  Dynamics 

4 

4 

Materials 

1 

0 
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Table  B.15 


Frequencies  of  Methods  Used  (Ranking  not  performed) 


Acrony 

ms: 

CPC 

Checklist  and  Profile  Charts 

SM 

Scoring  Models 

CBR 

Cost/Benefit  Ratio 

DT 

Decision  Trees 

LP 

Linear  Programming 

GP 

Goal  Programming 

DP 

Dynamic  Progranming 

CCP 

Chance  Constrained  Programming 

MOM 

Multiple  Objective  Method 

OTH 

Other 

Table  B.15-A 


Technique 

Frequency  of 
Response 

CPC 

5 

SM 

5 

CBR 

4 

DT 

5 

LP 

0 

GP 

5 

DP 

1 

CCP 

0 

MOM 

2 

OTH 

11 
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FDM 

Use 


FDM  vs  Techniques  Used 

CPC  SM  CBR  DT  LP  GP  DP  CCP  MOM  OTH 


Yes 

4 

4 

4  5 

0  5 

l 

0  2 

8 

No 

1 

1 

0  0 

0  0 

0 

0  0 

3 

Table 

B. 15-C 

FDM 

Management 

Science 

Background 

vs 

Techniques 

Used 

Use 

CPC 

SM 

CBR  DT 

LP  GP 

DP 

CCP  MOM 

OTH 

Yes 

1 

1 

2  2 

1 

0  1 

3 

No 

4 

4 

2  3 

HH 

0 

0  1 

8 

TABLE  B.  16 

Dollar  Values  of  the  Research  Projects 


Project 

Frequency  of 

Value 

Response 

Less  than  100K 

10 

100K  -  300K 

14 

300K  -  1M 

10 

More  than  1M 

6 
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TABLE  B. 17 


Selection  Techniques  Participants 
Were  Aware  Of 


Technique 

Frequency 
of  Awareness 

CPC 

18 

SM 

17 

CBR 

28 

DT 

28 

LP 

18 

GP 

14 

DP 

6 

CCP 

5 

MOM 

10 

OTH 

3 

Table  B.17-A 


CPC 

SM 

CBR 

Laboratory  Response 
DT  LP  GP  DP 

CCP 

Avionics 

5 

5 

8 

8 

7 

6 

2 

2 

Aero  Prop. 

6 

4 

6 

5 

4 

2 

0 

1 

Fit.  Dynamics 

5 

6 

8 

9 

4 

3 

3 

2 

Materials 

2 

2 

6 

6 

3 

3 

1 

0 

Table  B.17-B 


9# 

-I? 

4 


Researchers 


Supervisor 


Branch  Chief 


Directorate  Hd.  1 


Management  Response 

CPC  SM  CBR  DT  LP  GP  DP  CCP  MOM  OTH 


6  1 


7  1 


TABLE  B. 18 


Respondent's  Desire  to  Change  Selection  Methods 


Laboratory 


Avionics 


Aero  Propulsion 
Flight  Dynamics 


Materials 


Response 


Table  B.18-B 


Management  Response 

Researcher  2 
Supervisors  3 
Branch  Chiefs  0 
Directorate  Head  0 


> 


TABLE  B. 19 

Selection  Method  Varies  by  Project  Dollar  Value 


Overall  Response 


Yes 

12 

No 

28 

Table  B.19-A 


Positive 

Laboratory  Response 


Avionics 
Aero  Propulsion 
Flight  Dynamics 


5 

1 

5 


Materials 


1 


Table  B.19-B 


Management  Level 

Positive 

Response 

Researcher 

4 

Supervisor 

5 

Branch  Chief 

2 

Directorate  Head 

1 

TABLE  B.20 

Selection  of  Research  Project 

Portfolios 

Number  of  Respondents 

Who  Select  Portfolios 

Yes 

19 

No 

21 

Table  B.20-A 

Technique 

Response 

Linear  Programming 

1 

Goal  Programming 

5 

Multiple  Objective 

4 

Chance  Constrained  Prog 

0 

Other 


11 
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